04:00
\[\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix}\]
Rows are horizontal ↔︎
Columns are vertical ↕
\[ \mathbf{A} = \begin{bmatrix} \require{color}\colorbox{#eec8e1}{$a_{11}$} & \colorbox{#eec8e1}{$a_{12}$} & \colorbox{#eec8e1}{$a_{13}$} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \leftarrow \text{Row 1} \]
Rows are horizontal ↔︎
Columns are vertical ↕
\[ \mathbf{A} = \begin{bmatrix} \require{color}a_{11} & a_{12} & a_{13} \\ \colorbox{#eec8e1}{$a_{21}$} & \colorbox{#eec8e1}{$a_{22}$} & \colorbox{#eec8e1}{$a_{23}$} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \leftarrow \text{Row 2} \]
Rows are horizontal ↔︎
Columns are vertical ↕
\[ \mathbf{A} = \begin{bmatrix} \require{color}a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ \colorbox{#eec8e1}{$a_{31}$} & \colorbox{#eec8e1}{$a_{32}$} & \colorbox{#eec8e1}{$a_{33}$} \end{bmatrix} \leftarrow \text{Row 3} \]
Rows are horizontal ↔︎
Columns are vertical ↕
\[ \mathbf{A} = \begin{bmatrix} \colorbox{#eec8e1}{$a_{11}$} & a_{12} & a_{13} \\ \colorbox{#eec8e1}{$a_{21}$} & a_{22} & a_{23} \\ \colorbox{#eec8e1}{$a_{31}$} & a_{32} & a_{33} \end{bmatrix} \leftarrow \text{Column 1} \]
A matrix with n rows and p columns is called an n × p matrix
\[\mathbf{A}_{3 \times 3} = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix}\]
This is a 3 × 3 matrix (3 rows, 3 columns)
A matrix with n rows and p columns is called an n × p matrix
\[\mathbf{B}_{2 \times 4} = \begin{bmatrix} b_{11} & b_{12} & b_{13} & b_{14} \\ b_{21} & b_{22} & b_{23} & b_{24} \end{bmatrix}\]
This is a 2 × 4 matrix (2 rows, 4 columns)
Row vector (1 × p): \[\mathbf{r} = \begin{bmatrix} 1 & 3 & 5 & 7 \end{bmatrix}\]
Column vector (n × 1): \[\mathbf{c} = \begin{bmatrix} 2 \\ 4 \\ 6 \end{bmatrix}\]
Square matrix (n × n): \[\mathbf{S} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\]
Matrices can be added only if they have the same dimensions
Add corresponding elements: \[\mathbf{A} + \mathbf{B} = \begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\ a_{21} + b_{21} & a_{22} + b_{22} \end{bmatrix}\]
Example: \[\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}\]
\[= \begin{bmatrix} \color{red}{1+5} & \color{blue}{2+6} \\ \color{green}{3+7} & \color{purple}{4+8} \end{bmatrix} = \begin{bmatrix} \color{red}{6} & \color{blue}{8} \\ \color{green}{10} & \color{purple}{12} \end{bmatrix}\]
Problem: Add these matrices \(\mathbf{A} = \begin{bmatrix} 3 & -1 & 2 \\ 0 & 4 & -3 \end{bmatrix}, \quad \mathbf{B} = \begin{bmatrix} 1 & 2 & -1 \\ 5 & -2 & 1 \end{bmatrix}\)
You Try: \(\mathbf{A} + \mathbf{B} = ?\)
04:00
This works ✓
\[\colorbox{#eec8e1}{$\mathbf{A}_{2 \times 3} + \mathbf{B}_{2 \times 3} = \mathbf{C}_{2 \times 3}$}\]
This doesn’t work ✗
\[\colorbox{#eec8e1}{$\mathbf{A}_{2 \times 3} + \mathbf{B}_{3 \times 2} = \text{?}$}\]
The matrices must have exactly the same dimensions to be added!
Problem: Can these matrices be multiplied? If yes, what’s the resulting dimension?
\(\mathbf{A}_{3 \times 2} \times \mathbf{B}_{2 \times 4} = ?\)
\(\mathbf{C}_{2 \times 5} \times \mathbf{D}_{3 \times 2} = ?\)
Solutions:
\(\mathbf{A}_{3 \times 2} \times \mathbf{B}_{2 \times 4} = \mathbf{Result}_{3 \times 4}\) ✓ (inner dimensions match: 2 = 2)
\(\mathbf{C}_{2 \times 5} \times \mathbf{D}_{3 \times 2}\) = undefined ✗ (inner dimensions don’t match: 5 ≠ 3)
02:00
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}&\\&\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{}&\\&\end{bmatrix} \]
\[\begin{bmatrix} \colorbox{#eec8e1}{1} & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}{5} & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{(1)(5)}&\\&\end{bmatrix}\]
\[\begin{bmatrix} \colorbox{#eec8e1}{1} & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}{5} & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{(1)(5) + (2)(7)}&\\&\end{bmatrix}\]
\[\begin{bmatrix} \colorbox{#eec8e1}{1} & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}{5} & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}\colorbox{#eec8e1}{19}&\\&\end{bmatrix}\]
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{ }\\&\end{bmatrix} \]
\[ \begin{bmatrix} \colorbox{#eec8e1}1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{(1)(6)}\\&\end{bmatrix} \]
\[ \begin{bmatrix} \colorbox{#eec8e1}1 & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{(1)(6)+(2)(8)}\\&\end{bmatrix} \]
\[ \begin{bmatrix} \colorbox{#eec8e1}1 & \colorbox{#eec8e1}2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&\colorbox{#eec8e1}{22}\\&\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{ }&\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & 4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{(3)(5) }&\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}5 & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{(3)(5) +(4)(7)}&\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} \colorbox{#eec8e1}5 & 6 \\ \colorbox{#eec8e1}7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\\colorbox{#eec8e1}{43}&\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{ }\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & 8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{(3)(6) }\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{(3)(6)+(4)(8) }\end{bmatrix} \]
\[ \begin{bmatrix} 1 & 2 \\ \colorbox{#eec8e1}3 & \colorbox{#eec8e1}4 \end{bmatrix} \times \begin{bmatrix} 5 & \colorbox{#eec8e1}6 \\ 7 & \colorbox{#eec8e1}8 \end{bmatrix}=\begin{bmatrix}19&22{}\\43&\colorbox{#eec8e1}{52}\end{bmatrix} \]
Problem: Multiply these matrices \(\mathbf{A} = \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix}, \quad \mathbf{B} = \begin{bmatrix} 1 & 0 \\ 2 & 1 \end{bmatrix}\)
You Try: \(\mathbf{A} \times \mathbf{B} = ?\)
Solution:
\(\mathbf{A} \times \mathbf{B} = \begin{bmatrix} 8 & 3 \\ 9 & 4 \end{bmatrix}\)
05:00
Valid multiplications:
\(\mathbf{A}_{2 \times 3} \times \mathbf{B}_{3 \times 4} = \mathbf{C}_{2 \times 4}\) ✓
\(\mathbf{A}_{5 \times 2} \times \mathbf{B}_{2 \times 1} = \mathbf{C}_{5 \times 1}\) ✓
Invalid multiplications:
\(\mathbf{A}_{2 \times 3} \times \mathbf{B}_{4 \times 2}\) ✗ (3 ≠ 4)
\(\mathbf{A}_{3 \times 5} \times \mathbf{B}_{2 \times 3}\) ✗ (5 ≠ 2)
Remember: Inner dimensions must match!
The transpose of matrix \(\mathbf{A}\) is denoted \(\mathbf{A}^T\) or \(\mathbf{A}'\)
Flip rows and columns
\[\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}_{2 \times 3}\]
\[\mathbf{A}^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}_{3 \times 2}\]
Find the transpose of this matrix \(\mathbf{A} = \begin{bmatrix} 1 & 4 & 7 \\ 2 & 5 & 8 \end{bmatrix}\) \(A^T= ?\)
Solution: \(\mathbf{A}^T = \begin{bmatrix} 1 & 2 \\ 4 & 5 \\ 7 & 8 \end{bmatrix}\)
03:00
If \(\mathbf{A}\) is \(m \times n\), then \(\mathbf{A}^T\) is \(n \times m\)
\((\mathbf{A}^T)^T = \mathbf{A}\)
\((\mathbf{A} + \mathbf{B})^T = \mathbf{A}^T + \mathbf{B}^T\)
\((\mathbf{AB})^T = \mathbf{B}^T\mathbf{A}^T\) (order reverses!)
Symmetric Matrix: \(\mathbf{A} = \mathbf{A}^T\)
\(\mathbf{A} = \mathbf{A}^T\)
\[\mathbf{S} = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 5 \\ 3 & 5 & 6 \end{bmatrix}\]
The identity matrix \(\mathbf{I}\) is the matrix equivalent of the number 1
Square matrix with 1’s on the diagonal and 0’s elsewhere
Acts as the multiplicative identity: \(\mathbf{AI} = \mathbf{IA} = \mathbf{A}\)
\[\mathbf{I}_2 = \begin{bmatrix} \color{red}{1} & 0 \\ 0 & \color{red}{1} \end{bmatrix}, \quad \mathbf{I}_3 = \begin{bmatrix} \color{red}{1} & 0 & 0 \\ 0 & \color{red}{1} & 0 \\ 0 & 0 & \color{red}{1} \end{bmatrix}\]
Example: \[\begin{bmatrix} 2 & 3 \\ 4 & 5 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 3 \\ 4 & 5 \end{bmatrix}\]
Problem: Verify that this multiplication gives the identity property \(\begin{bmatrix} 3 & -1 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = ?\)
Solution: \(= \begin{bmatrix} 3 & -1 \\ 2 & 4 \end{bmatrix}\) ✓ (Original matrix unchanged!)
03:00
The inverse of matrix \(\mathbf{A}\) is denoted \(\mathbf{A}^{-1}\)
Definition: \(\mathbf{AA}^{-1} = \mathbf{A}^{-1}\mathbf{A} = \mathbf{I}\)
Requirements for inverse to exist:
- Matrix must be square (n × n)
- Matrix must be non-singular (determinant ≠ 0)
2 × 2 Formula: For \(\mathbf{A} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\)
\[\mathbf{A}^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}\]
\[\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix}\]
Step 1: Calculate determinant \[\text{det}(\mathbf{A}) = (2)(2) - (1)(6) = 4 - 6 = -2\]
\[\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix}\]
Step 2: Apply formula \[\mathbf{A}^{-1} = \frac{1}{2} \begin{bmatrix} 2 & -1 \\ -6 & 2 \end{bmatrix} = \begin{bmatrix} 1 & -0.5 \\ -3 & 1 \end{bmatrix}\]
\[\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix}\]
Verify: \[\mathbf{AA}^{-1} = \begin{bmatrix} 2 & 1 \\ 6 & 2 \end{bmatrix} \begin{bmatrix} 1 & -0.5 \\ -3 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \mathbf{I}\]
Problem: Find the inverse of this matrix \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 4 & 3 \end{bmatrix}\)
Solution: 1. \(\text{det}(\mathbf{A}) = (1)(3) - (2)(4) = 3 - 8 = -5\)
2. \(\mathbf{A}^{-1} = \frac{1}{-5} \begin{bmatrix} 3 & -2 \\ -4 & 1 \end{bmatrix} = \begin{bmatrix} -0.6 & 0.4 \\ 0.8 & -0.2 \end{bmatrix}\)
Check: \(\mathbf{AA}^{-1} = \begin{bmatrix} 1 & 2 \\ 4 & 3 \end{bmatrix} \begin{bmatrix} -0.6 & 0.4 \\ 0.8 & -0.2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\) ✓
06:00
Diagonal Matrix: Non-zero elements only on the main diagonal
\[\mathbf{D} = \begin{bmatrix} d_1 & 0 & 0 \\ 0 & d_2 & 0 \\ 0 & 0 & d_3 \end{bmatrix}\]
Orthogonal Matrix: \(\mathbf{Q}^T\mathbf{Q} = \mathbf{I}\) (columns are orthonormal)
Positive Definite: \(\mathbf{x}^T\mathbf{A}\mathbf{x} > 0\) for all \(\mathbf{x} \neq \mathbf{0}\)
Symmetric Matrix: \(\mathbf{A} = \mathbf{A}^T\)
Problem: Identify the type of each matrix \(\mathbf{A} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 3 \end{bmatrix}, \quad \mathbf{B} = \begin{bmatrix} 1 & 4 \\ 4 & 2 \end{bmatrix}, \quad \mathbf{C} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}\)
Solutions: - Matrix A: Diagonal matrix (non-zero elements only on main diagonal) - Matrix B: Symmetric matrix (\(\mathbf{B} = \mathbf{B}^T\) since \(b_{12} = b_{21} = 4\)) - Matrix C: General matrix (no special properties)
02:00
Why do we need derivatives of vectors? - Essential for optimization in statistics and machine learning - Key to deriving least squares solutions - Foundation for understanding the hat matrix
Let \(\mathbf{a} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix}\) and \(\mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix}\) be column vectors
Rule: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{a}'\mathbf{x}) = \mathbf{a}\)
In words: The derivative of a linear form \(\mathbf{a}'\mathbf{x}\) with respect to \(\mathbf{x}\) is simply \(\mathbf{a}\)
Let \(\mathbf{a} = \begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix}\) and \(\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}\)
Then \(\mathbf{a}'\mathbf{x} = 2x_1 + 3x_2 + x_3\)
Derivative: \(\frac{\partial}{\partial \mathbf{x}}(2x_1 + 3x_2 + x_3) = \begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix} = \mathbf{a}\)
Problem: Find \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{c}'\mathbf{x})\) where \(\mathbf{c} = \begin{bmatrix} 5 \\ -2 \\ 4 \end{bmatrix}\)
Solution: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{c}'\mathbf{x}) = \mathbf{c} = \begin{bmatrix} 5 \\ -2 \\ 4 \end{bmatrix}\)
03:00
Quadratic form: \(\mathbf{x}'\mathbf{A}\mathbf{x}\) where \(\mathbf{A}\) is an \(n \times n\) matrix
Example: \(\mathbf{x}'\mathbf{A}\mathbf{x} = \begin{bmatrix} x_1 & x_2 \end{bmatrix} \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\)
Expanding the 2×2 case: \[\mathbf{x}'\mathbf{A}\mathbf{x} = a_{11}x_1^2 + (a_{12} + a_{21})x_1x_2 + a_{22}x_2^2\]
Rule: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}')\mathbf{x}\)
Special case: If \(\mathbf{A}\) is symmetric (\(\mathbf{A} = \mathbf{A}'\)), then: \[\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = 2\mathbf{A}\mathbf{x}\]
Let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}\) and \(\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\)
Step 1: Find \(\mathbf{A} + \mathbf{A}'\) \[\mathbf{A}' = \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix} \mathbf{A} + \mathbf{A}' = \begin{bmatrix} 4 & 4 \\ 4 & 8 \end{bmatrix}\]
Let \(\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}\) and \(\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\)
Step 2: Apply the rule \[\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}')\mathbf{x} = \begin{bmatrix} 4 & 4 \\ 4 & 8 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 4x_1 + 4x_2 \\ 4x_1 + 8x_2 \end{bmatrix}\]
Example: Let \(\mathbf{S} = \begin{bmatrix} 3 & 2 \\ 2 & 5 \end{bmatrix}\) (symmetric)
Derivative: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{S}\mathbf{x}) = 2\mathbf{S}\mathbf{x} = 2\begin{bmatrix} 3 & 2 \\ 2 & 5 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 6x_1 + 4x_2 \\ 4x_1 + 10x_2 \end{bmatrix}\)
Rule: \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{b}'\mathbf{A}\mathbf{x}) = \mathbf{A}'\mathbf{b}\)
Note: \(\mathbf{b}'\mathbf{A}\mathbf{x}\) is a scalar, and \(\mathbf{b}\) is treated as a constant vector
Problem: Find \(\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{B}\mathbf{x})\) where \(\mathbf{B} = \begin{bmatrix} 1 & 0 \\ 2 & 3 \end{bmatrix}\)
Solution: \[\mathbf{B}' = \begin{bmatrix} 1 & 2 \\ 0 & 3 \end{bmatrix}, \quad \mathbf{B} + \mathbf{B}' = \begin{bmatrix} 2 & 2 \\ 2 & 6 \end{bmatrix}\] \[\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}'\mathbf{B}\mathbf{x}) = \begin{bmatrix} 2 & 2 \\ 2 & 6 \end{bmatrix} \mathbf{x} = \begin{bmatrix} 2x_1 + 2x_2 \\ 2x_1 + 6x_2 \end{bmatrix}\]
04:00
These rules are essential for deriving:
Normal equations: \(\mathbf{X}'\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}'\mathbf{y}\)
Hat matrix: \(\mathbf{H} = \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\)
Least squares solution: \(\hat{\boldsymbol{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{y}\)
We’ll use these derivatives to minimize the sum of squared errors!